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Abstract 

Background: Reference management software programs enable researchers to more easily organize and manage 
large volumes of references typically identified during the production of systematic reviews. The purpose of this 
study was to determine the extent to which authors are using reference management software to produce 
systematic reviews; identify which programs are used most frequently and rate their ease of use; and assess the 
degree to which software usage is documented in published studies. 

Methods: We reviewed the full text of systematic reviews published in core clinical journals indexed in ACP Journal 
Club from 2008 to November 201 1 to determine the extent to which reference management software usage is 
reported in published reviews. We surveyed corresponding authors to verify and supplement information in 
published reports, and gather frequency and ease-of-use data on individual reference management programs. 

Results: Of the 78 researchers who responded to our survey, 79.5% reported that they had used a reference 
management software package to prepare their review. Of these, 4.8% reported this usage in their published 
studies. EndNote, Reference Manager, and RefWorks were the programs of choice for more than 98% of authors 
who used this software. Comments with respect to ease-of-use issues focused on the integration of this software 
with other programs and computer interfaces, and the sharing of reference databases among researchers. 

Conclusions: Despite underreporting of use, reference management software is frequently adopted by authors of 
systematic reviews. The transparency, reproducibility and quality of systematic reviews may be enhanced through 
increased reporting of reference management software usage. 

Keywords: Meta-analysis, Systematic reviews, Data collection, Databases, Bibliographic, Software, Reference 
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Background 

Various software programs have been adopted by, and spe- 
cifically developed for, authors of systematic reviews. 
These tools range from statistical software to comprehen- 
sive systematic reviews programs, such as the Cochrane 
Collaboration's Review Manager (RevMan) software. Ref- 
erence management software programs, including End- 
Note, RefW orks and Zotero, are one such tool. 

Reference management software, also known as biblio- 
graphic software, citation management software, or per- 
sonal bibliographic file managers is "any software product 
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used for storage and retrieval of bibliographic records" [1]. 
First developed in the 1980s, these programs were initially 
marketed to researchers as a means of creating online in- 
dexes of personal print-article collections [2-4]. As elec- 
tronic databases, such as MEDLINE, became generally 
accessible and more easily searchable, researchers began 
to use this software to maintain databases of all research 
relevant to their fields of interest [3]. 

Numerous reference management programs are cur- 
rently available. Although all programs facilitate the cap- 
ture, organization, and elimination of duplicate records 
from electronic database searching, they vary with respect 
to cost, overall functionality, and networking capabilities. 
Products such as EndNote, Papers, and RefWorks are 
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licensed or sold outright, while others (e.g.: Mendeley and 
Zotero), are available at little or no cost to the user. 
While some, such as EndNote and Reference Manager, run 
on single-station computers, many others, including Ref- 
Works, Mendeley and Zotero, are web-based. Single- 
station software usage is generally not affected by website 
time-lags, down times, or record limits, all of which may 
impinge on the usability of web-based products. That said, 
the benefits of these web-based programs include the abil- 
ity to store reference databases on secure servers, and ac- 
cess databases from multiple computers or other electronic 
devices. Web-based programs also provide users with en- 
hanced networking functions that readily support the shar- 
ing of records among researchers [3,5] . 

The identification, collection, and organization of rele- 
vant studies are instrumental to the successful completion 
of systematic reviews. To this end, a "rigorous data man- 
agement plan" is essential [6]. Many guides to undertaking 
systematic reviews, including the Cochrane Handbook for 
Systematic Reviews of Interventions, recommend reference 
management software as a means of assisting in the or- 
ganization and selection of component studies for inclu- 
sion in these reviews [6,7]. 

In the context of systematic reviews, reference manage- 
ment programs facilitate the capture and organization of 
studies identified through electronic database searching, 
the identification and elimination of duplicate records 
from multiple database searches, the transfer of references 
to Cochrane RevMan and other systematic reviews soft- 
ware, and the accurate citing of references within manu- 
scripts [8,9]. Thus, an author's decision to use, or not use, 
this software may impact on the accurate reporting of the 
number of studies reviewed for inclusion and exclusion in 
a systematic review. 

The purpose of this study was to determine the extent 
to which authors are using reference management soft- 
ware to produce systematic reviews; identify which pro- 
grams are used most frequently and rate their ease of 
use; and assess the degree to which software usage is 
documented in published studies. 

Methods 

We reviewed the full text of systematic reviews published 
in core clinical journals. We surveyed the corresponding 
authors of included studies to verify and supplement infor- 
mation in published reports, and gather frequency and 
ease-of-use data on specific reference management pro- 
grams. This study received ethics approval from the 
University of Calgary's Conjoint Health Ethics Board. 

Study identification 

We retrieved the full-text reports of all systematic re- 
views indexed in the ACP Journal Club from 2008 to 
November 2011. ACP Journal Club indexes over 100 



core clinical journals. Our decision to study systematic 
review articles featured in ACP Journal Club relates par- 
tially to the clinical importance and visibility of these 
featured articles. We wanted our study to focus on 
articles that the scientific community has judged to be 
important. Systematic reviews indexed in ACP Journal 
Club must meet standard quality criteria including a 
clear statement of research, and a description of the 
methods used to identify and select studies for inclusion 
in the review. By restricting our data collection to stud- 
ies indexed in ACP Journal Club, we recognize that we 
have been sampling reviews with better-than-average 
reporting. 

Studies were included if they were English language sys- 
tematic reviews or meta-analyses. Studies were excluded if 
they were: (1) meta-analyses of data not obtained through 
reviews of the published literature (e.g.: chart reviews); 
(2) systematic reviews where only 1 electronic database 
was searched to identify relevant studies; or (3) publica- 
tions where the email addresses of corresponding authors 
were not provided, or could not be determined. We 
searched PubMED, Scopus, Web of Knowledge, Google, 
and Google Scholar to identify current email addresses for 
corresponding authors of studies selected for review. 

Data collection 

We undertook a content analysis of included reviews to 
identify those authors who had reported on the use of 
reference management software in their published re- 
views. As a counterpoint, we also noted the frequency of 
reporting of statistical software usage. An Excel form 
was created to extract the following data from each 
study: author, date, title, source, corresponding author's 
name and email address, and documented use of refer- 
ence management and statistical software. Where au- 
thors were listed as corresponding authors in more than 
one publication, only the most recent study was selected 
for review. 

A survey was emailed to all corresponding authors to 
compare actual with reported usage, and gather data on 
specific software programs. The five-item survey asked 
authors to: (1) indicate whether or not a reference man- 
agement program had been used; (2) provide the name 
of the program; (3) rate its functionality on a scale of 0 
to 10, with 0 being the "least functional" and 10 the 
"most functional"; (4) indicate their intention to use this 
product in future reviews; and (5) provide any additional 
comments on the usability of this software. Two re- 
minders were sent to corresponding authors at one- 
month intervals. 

Data analysis 

Descriptive statistics were calculated using SPSS V20 
statistical analysis software. Responses to the final open- 
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ended survey question (Q#5) were analyzed to identify 
common themes. 

Results 

From a total of 163 systematic reviews identified in ACP 
Journal Club, 111 authors, representing 122 papers, were 
included in this study. Based on eligibility criteria, 41 pa- 
pers were excluded from this study. Reasons for exclu- 
sion were: an inability to determine or verify the email 
address of the corresponding author (n = 30); study au- 
thors only searched 1 electronic database to identify 
studies to include in their systematic review (n = 8), or 
no electronic databases were searched (n = 3). Of the 
111 surveys that were emailed to authors, 78 surveys 
were returned, for a response rate of 70.27%. 

A total of 79.5% (62/78) of respondents indicated that 
they had used a reference management software pro- 
gram in preparing their reviews. Of these, 4.8% (3/78) 
had included this information in their published reviews. 
In comparison, 76.9% (60/78) of authors reported on the 
use of statistical software in their published reviews 
(Table 1). 

The reference management software program used most 
frequently was EndNote (n = 41), followed by Reference 
Manager (n = 14), RefWorks (n = 6) and Excel (n=l). 
Mean functionality ratings for these tools ranged from 7.0 
to 8.0 (Table 2). 

Of the 62 authors who reported using reference man- 
agement software, 4 (2 EndNote and 2 Reference Man- 
ager) stated that they would be switching to a different 
software package in future reviews. Stated reasons for 
switching software packages included: incompatibility is- 
sues between Reference Manager and Mac computers, 
difficulty importing references into EndNote, and, in one 
instance, a cost savings that would result from replacing 
EndNote with Zotero. 

Forty-seven authors (60.3%) responded to our final 
open-ended survey question asking for comments on the 
usability of reference management programs. Thirteen au- 
thors remarked on the ease of use and overall importance 
of their chosen tool. As one respondent stated: 

"I don't think you can write manuscripts for 
prestigious journals without one of these bibliography 

management programs there will always be 

recommended revisions, or a rejection... in which 
case you'll have to reconfigure your references. To do 



[this] by hand is too cumbersome and time- 
consuming". 

A number of authors also identified specific challenges 
associated with these programs. Among these were: record 
errors that occurred when downloading references from 
electronic databases, such as MEDLINE (n = 4), difficulties 
in identifying and deleting duplicate records from reference 
management databases (n = 2); PC/Mac-incompatibilities 
for users of EndNote and Reference Manager (n = 3); er- 
rors in journal output styles (n = 3); difficulties in transfer- 
ring reference databases from one software package to 
another (n = 2), and delays in accessing RefWorks data- 
bases (n = 2). Five respondents indicated that they were not 
only using this software to collate studies identified 
through database searching, but also to record reviewers' 
decisions with respect to the inclusion or exclusion of 
studies. 

Discussion 

Our study reveals that most (but not all) authors of sys- 
tematic reviews are using reference management soft- 
ware tools. Despite this, only a small minority are 
reporting this usage in their published studies. Further, 
there appears to be no apparent relationship between 
choice of reference management software and perceived 
functionality or usability. Nor was software choice asso- 
ciated with the degree to which usage was reported or 
not reported in published studies. 

Currently, there are more than 28,000 scholarly journals 
in active publication [10]. Collectively, these journals pub- 
lish approximately 1.8 million articles per year [10]. Given 
the trend, across disciplines, towards integrating evidence 
into daily practice, it is unsurprising that an increasing 
number of systematic reviews are being funded and pro- 
duced. Previous research has demonstrated that authors 
typically search multiple electronic databases to identify 
studies relevant for systematic reviews [11-14]. Estimates 
of overlap between these databases have ranged between 
8% and 60% [11-14]. Software programs that can facilitate 
the collection, organization and removal of duplicate stud- 
ies from database searches are invaluable in the produc- 
tion of large-scale systematic reviews. 

The most significant finding from this study was the 
disparity between stated and reported software use. Few 
(4.8%) authors who had used reference management 
software reported this information in their published 



Table 1 Software usage and reporting among 78 respondents 





Reference management software 
frequency 


Reference management software 
percent 


Statistical software 
frequency 


Statistical software 
percent 


Usage 


62 


79.5% 


78 


100% 


Reported 


3 


4.8% 


60 


76.9% 
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Table 2 Reference software usage by type and 



functionality among 78 respondents 





Frequency 


Percent 


Mean functionality rating 


EndNote 


41 


52.6% 


7.5 (range 3 to 10) 


Reference Manager 


14 


1 7.9% 


7.0 (range 6 to 9) 


RefWorks 


6 


7.7% 


7.8 (range 6 to 10) 


Excel 


1 


1 .3% 


8.0 


None 


16 


20.5% 


NA 


Total 


78 


1 00% 





manuscripts. As a point of contrast, 76.9% of the authors 
in our sample reported on the statistical software pack- 
ages used to analyze study results. A recent study on the 
use of statistical software in health services research also 
found that a significant number of authors were includ- 
ing statistical software information in their published 
manuscripts [15]. This discrepancy in reporting may be 
the result of an absence of guidelines which specifically 
address this aspect of reporting. Whereas The Inter- 
national Committee of Medical Journal Editors' Uniform 
Requirements for Manuscripts Submitted to Biomedical 
Journals recommend that authors "specify the computer 
software" when describing the statistical methods they 
employed to analyze study data, no mention is made, 
here or elsewhere, of the importance of reporting on the 
use of reference management software [16]. 

Although the PRISMA checklist for the quality report- 
ing of systematic reviews does not specifically suggest that 
authors report on reference management software usage, 
its use can impact the number of unique studies identified 
and reported in a systematic review [17]. A failure to iden- 
tify duplicate records captured in database searches may 
result in an over-reporting of irrelevant studies. Similarly, 
an over-reliance on reference management software to 
identify and remove duplicate records from a reference 
database may cause relevant studies to be overlooked. As 
such, reference management software can have a real im- 
pact on the quality of a completed review. 

Whereas numerous studies have been published citing 
the benefits of reference management and other sofware 
programs in the production of systematic reviews, the de- 
gree to which these programs have been adopted by the 
research community is largely unknown [2,3,6,9,18-22]. In 
2007, Senarath published a research letter on the reference 
management software experiences of 22 researchers in Sri 
Lanka [8]. Although informative in terms of highlighting 
the functions and perceived advantages of this software, to 
our knowledge, ours is the first study to explore reference 
management software usage and reporting among authors 
of systematic reviews. 

Our study has caveats and limitations. Although we 
were able to achieve a strong response rate of 70% from 
our email survey, a telephone survey could have yielded 



a higher response rate. Furthermore, researchers who 
declined to participate in our study may have different 
perspectives on reference management software usage 
than those who participated. Secondly, as mentioned 
previously, we limited our study sample to clinical re- 
views published in ACP Journal Club. This sampling 
frame may have resulted in the identification of pub- 
lished systematic reviews that have a better-than-average 
quality of reporting. Thirdly, an analysis of systematic re- 
views produced in languages other than English, and 
other disciplines, such as education or social work, could 
yield different findings. Finally, with the exception of 
one open-ended question, our survey did not seek to 
identify perceived benefits and deficits with respect to 
individual software programs. 

An overriding caveat to our study, and its implications 
in terms of the importance of reporting reference man- 
agement software usage, is that there is, as yet, no firm 
evidence to suggest that reference management software 
can have a positive impact on the quality of completed 
reviews. That said, the "value of a systematic review de- 
pends on what was done, what was found, and the clar- 
ity of reporting" [17]. As stated by Moher and colleagues 
"the reporting and conduct of systematic reviews are, 
by nature, closely intertwined" [17]. As such, a detailed 
transparent plan for the identification and selection of 
component studies, including the use or lack of use of 
reference management software, is an important element 
of any review protocol. It can reflect the methodological 
expertise of the authors, and the transparency, reprodu- 
cibility and, ultimately, the quality of the study that has 
been produced. 

Conclusions 

Despite being underreported, reference management 
software is frequently used by authors of systematic re- 
views. The transparency, reproducibility and quality of 
systematic reviews may be enhanced through increased 
reporting of reference management software usage. 
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